Search for the Lepton Flavour Violating Higgs decay H ^ rfx at Hadron Colliders 

U. Cotti"j] L. Diaz-Cruz''J] C. Pagliarone^J] and E. Vatagaf] 

a)IFM-UMSNH, A. P. 2-82, 58041 Morelia, Mexico 
b) Instituto de Fisica, Benemerita Universidad Autonoma de Puebla, 
Ap. Postal J-48, 72500, Puebla, Pue., Mexico and 
c) IN FN, Pisa, Italy 

We study the prospects to detect at hadron coUiders the Lepton Flavour Violating Higgs decay 
H — > T^L, which can reach substantial branching fractions in several extensions of the SM. Among 
them, the generic two higgs doublet model (THDM-III) can be taken as a representative case where 
B.R.{II Tn) can reach values of order ~ 10"^ — 10^^. Bounds on the LFV factor of order 
0.8 - 1.7 can be derived at 95 % c.l. at Tevatron Run-2 with 4 fb'^ for tuh = 110 - 150 GeV. 
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CN . I. INTRODUCTION. 

> : 

O ■ The search for the Higgs boson is one of the main goals of Tevatron RUN-2 and future coUiders 0| . Although 
I the most conservative search strategy uses the theoretical expectations coming from the minimal standard model 
. (SM), it is certainly worthwhile to look for other signals arising from physics beyond the SM. In this regard, it 
' has been recognized recently that the Higgs sector of several well-motivated models can predict lepton flavour 
violating (LFV) Higgs decays at sizable rates that may be detectable at future colliders 1^, In fact, 
T-H ' violation of Lepton number is predicted in several extensions of the standard model (SM), and the results on 
^ , atmospheric neutrinos showing evidence for neutrino oscillations, indicate that a lepton sector beyond the 
' SM is required to account for the pattern of neutrino masses and mixing suggested by the data. 
I In this paper we report preliminary work started at Snowmass 2001, regarding the detectability of such 
LFV Higgs decays at Hadron colliders. Although such decays were studied in |Q, within the context of several 
extensions of the SM, including both the effective lagrangian approach as well as specific models, we shall express 
our results following the general Two-Higgs doublet model (THDM-III), which does serves as a prototype model 
where such effects are predicted and it also facilitates the presentation of bounds on the LFV couplings. The 
main result of this work concerns the decays H ~* Tfi, which was found to reach a B.R. of order 0.1 — .01, for 
both the effective lagrangian and THDM-III cases, which can be searched at Tevatron Run-II. 
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II. THE LFV HIGGS DECAYS IN THE THDM-III 



The two-Higgs doublet extension of the SM (THDM), (Models I and II) solved their problem with large 
Flavour changing neutral scalar interactions (FCNSI), by requiring a discrete symmetry that restricted each 
^ ' fermion to couple at most to one Higgs doublet |Q. Later on, it was found that FCNSI could be suppressed at 
5^ 1 acceptable rates, with relatively light Higgs bosons, by imposing a more realistic pattern on the Yukawa matrices, 
which in principle can be associated with some family symmetry [Q. The phenomenological predictions of this 
model (called model HI in the literature ||, ^J) have been studied to some extent. 

Working within the Higgs mass-eigenstate basis, the LFV interactions of the light neutral Higgs boson H take 
the form: 

Clfv ^ cos alii jH + h.c. (1) 

where a denotes the mixing angle of the neutral Higgs sector, and Cij denotes the Yukawa coupling of the second 
doublet. In order to satisfy the low energy data on FCNC, Cheng and Sher ||l^, |l^ proposed the following 
ansatz : 

_ (m,mj)V^ 
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where v = 246 GeV and the lepton mass factor gives the order of magnitude of the interaction. The coefRcients 
Xij are dimensionless parameters that can be constrained by comparing the prediction for relevant processes 
with present experimental bounds on FCNC and LFV transitions. The strongest bound for the parameters Xij 
are obtained from muon anomalous magnetic moment namely: A^,- < 10, which involves only one coupling. 
Other interesting bounds are: (Ae^A^,-)^^^ < 5, which is obtained from the decay fj, 67. 

One additional implication of these LFV couplings is the possibility to observe the LFV Higgs decays H — » 
iflj , whose decay width will be given by: 

T{H ^ l+r) = cos^ a ruH (3) 

■' OTT 

Since the dominant decay mode of the Higgs boson in the intermediate mass range is — » bb, and the corre- 
sponding coupling will depend on rub, sin^ a and A^b, this will introduce a complicated expression for the Higgs 
total width. However, if we assume values for the parameters Ar^^ and sin a of order one, which satisfy present 
low-energy bounds, and neglect the corrections to all other higgs decays, we find that — > /ir is allowed to 
have a B.R. of order 0.1 — 0.01, which seems at the reach of future colliders. On the other hand, the resulting 
upper limit on B.R.(/i'^ e/i) is of the order 10^^ — 10^^, which does not seem to be at the reach of future 
experiments. Results for the LFV higgs decays H Tfj, and H — > re, are shown in table 1. Within the minimal 
SUGRA-MSSM and the SM with massive neutrinos, these decays are found to have negligible rates. Whereas 
in models with heavy majorana neutrinos, the LFV Higgs decays are induced at one-loop level and can reach 
values of order 10"^ ^ 



III. STRATEGY SEARCH FOR H ^ rfi 



In order to study these LFV Higgs decays at future hadron colliders, we shall focus on the mode H r/^, 
which has the muon in the final state and it is easier to separate from the backgrounds. Then, to derive bounds 
on the LFV parameters we shall consider the THDM-III, but for this it would be necessary to take into account 
not only the dependence of the Higgs decay width on these parameters, but also the full expression for the 
branching ratio. In order to handle such multi-parameter dependence, we shall introduce the parameter K^r, in 
such a way that the branching ratio for the decay H — > r/i, is given by: 

B.KiH ^ tm) = < (^) B.R.{H - r+r-) (4) 

where the dependence on the lepton masses has been made explicit, whereas the dependence on the parameters 
\ij and a has been absorbed into the couplings k^-^. 

In order to study the possibility to detect the LFV higgs decays, one can use the gluon-fusion mechanism to 
produce a single Higgs boson; assuming that the production cross-section is of similar strength to the SM case, 
about 1.2 pb for ran = 100 GeV, it will allow to produce 12,000 Higgs bosons with an integrated luminosity 
of 10 fb-^. Thus, for B.R.{H rn/re) ~ 10"^ - lO'^ Tevatron can produce 1200 - 120 events. Then, to 
determine the detectability of the signal, we need to study the main backgrounds to the H Tji signal, which 
are dominated by Drell-Yan tau pair and WW pair production. In Ref. Q it was proposed to reconstruct the 
hadronic and electronic tau decays, assuming the following cuts: i) For the transverse muon and jet momentum: 
> mi//5, > 10 GeV, ii) Jet rapidity for Tevatron (LHC): |r/| < 2(2.5) iii) The angle between the missing 
transverse momentum and the muon direction: (/)(/i±) > 160°. 

The resulting bounds on the LFV higgs couplings k.-^^ that can be obtained at Run-2 and LHC at 95% c.L, 
are shown in Fig. 0; one can see that it will be possible to test values of Urti of order 0.8 — 1.7 (0.2 — 0.4) at 
Tevatron (LHC) with 4 (100) fb~^ for mu = 110 - 130 GeV. In Fig. | we have also included the expected 
bound on k,-^ at the very large hadron collider (VLHC), with cm. energy of 40 TeV and integrated luminosity 
of 1000 fb^^, under the very crude assumption that signal and background can be scaled from LHC results; in 
this case the sensitivity extends up to values of k-t^ = 0.1. 

Thus, we found that the LFV Higgs decays H Tfi/re can have large branching ratios, of order 0.1 — 0.01, 
in some extensions of the SM, which can be detected at the coming stages of Tevatron Run-II. At present we 
are studying the signal at Tevatron Run-2 using the realistic detector simulation of CDF, which we expect to 
present in the near future mM. 
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TABLE I: B.R. of LFV Higgs decays for the THDM-III. Results are shown for sin a = 0.1, and the numbers in parenthesis 
correspond to sin a = 0.9. 
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FIG. 1: Bounds on the LFV coupUng n-rij, that can be obtained at Tevatron Run-2 (dot-dashes), LHC (dots) and VLHC 
(dashes) . 
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